Extracting layers and analyzing their specular properties using epipolar-plane-image analysis

Despite progress in stereo reconstruction and structure from motion, 3D scene reconstruction from multiple images still faces many difficulties, especially in dealing with occlusions, partial visibility, textureless regions, and specular reflections. Moreover, the problem of recovering a spatially dense 3D representation from many views has not been adequately treated. This document addresses the problems of achieving a dense reconstruction from a sequence of images and analyzing and removing specular highlights. The first part describes an approach for automatically decomposing the scene into a set of spatio-temporal layers (namely EPI-tubes) by analyzing the epipolar plane image (EPI) volume. The key to our approach is to directly exploit the high degree of regularity found in the EPI volume. In contrast to past work on EPI volumes that focused on a sparse set of feature tracks, we develop a complete and dense segmentation of the EPI volume. Two different algorithms are presented to segment the input EPI volume into its component EPI tubes. The second part describes a mathematical characterization of specular reflections within the EPI framework and proposes a novel technique for decomposing a static scene into its diffuse (Lambertian) and specular components. Furthermore, a taxonomy of specularities based on their photometric properties is presented as a guide for designing further separation techniques. The validity of our approach is demonstrated on a number of sequences of complex scenes with large amounts of occlusions 1077-3142/$ see front matter 2004 Elsevier Inc. All rights reserved. doi:10.1016/j.cviu.2004.06.001 * Corresponding author. Fax: +44-(0)1223-479999. E-mail address: [email protected] (A. Criminisi). 52 A. Criminisi et al. / Computer Vision and Image Understanding 97 (2005) 51–85 and specularity. In particular, we demonstrate object removal and insertion, depth map estimation, and detection and removal of specular highlights. 2004 Elsevier Inc. All rights reserved.